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ABSTRACT 

This paper argues that the scholarly review process 
in referred academic journal publishing restricts research creativity 
and timeliness, promotes inertia, and wastes resources. The 
publishing process of a Canadian journal (The Canadian Journal of 
Educational Communication), published three times annually, which 
uses a blind referee process is described as an example. The paper 
details a manuscript's journey from initial administrative check, to 
editorial review, to peer reviewer selection, to peer review, to 
publication decision, to return to author for suggested revisions, to 
final check. Benefits to this process are identified such as 
opportunities for valuable critical commentary helpful to other 
academics, sifting of the best material for publication, convenient 
method of discrimination for tenure evaluation, establishment of 
professional standards for research and scholarship, and keeping 
costs down. Drawbacks to peer review are also described and they 
include reviewer bias, discouragement of the exploration of fringe 
topics, and a generally slow pace of scholarly change. The long time 
taken by peer review also causes many articles to go unpublished. 
Finally the process of producing print-based journals is costly. The 
paper suggests that more journals move to alternative electronic 
formats which would allow publishing of everything submitted and 
rapid introduction of material into the academic marketplace. (JLj 
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Abstract 

This paper argues that the scholarly review process inevitably restricts research creativity and 
timeliness, promotes inertia, and wastes human and material resources. Electronic publishing 
is offered as one possible solution to several of the problems identified. 
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Some Reflections on Scholarly Review and Academic 

Publication 

This article has one simple argument to make: academic journals need to find a new way to do 
business. The scholarly review process many institutions seem to cherish stunts the growth of 
scholarship in many fields of study, and creates cumbersome administrative structures which 
are expensive and unnecessary. 

Why am I writing this? I have recently completed a term as editor of a major academic journal, 
The Canadian Journal of Educational Communication [CJEQ . The process was enlightening, 
and as I discussed several of my experiences with editors of other journals, I was surprised and 
relieved to learn that many spoke of the same joys and frustrations I encountered. I believe 
there is a growing concern that the typical academic structure no longer best serves our goals of 
promoting excellent scholarship, providing a forum for the exchange of ideas, and extending 
knowledge in our fields of study. 

I will aim my comments at refereed academic journals in the field of education because I am 
most familiar with these publications, but I sense that my criticisms and suggestions may 
apply equally to other fields. To provide a context for my arguments, I will briefly describe the 
review and publication process followed at GJEC. This process is fairly typical of other 
refereed academic journals in our field of study. 1 

A Typical Structure 



1 These included Educational Technology Research and Development, British Journal of 
Educational Technology, Performance Improvement Quarterly, Canadian Journal of 
Distance Education, and the Canadian Journal of Education. 
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The Canadian Journal of Educational Communication uses a blind referee process and it is 
indexed by the Educational Resources Information Center (ERIC) and the Canadian Education 
Index. It is published three times annually through the support of the Social Sciences and 
Humanities Research Council of Canada. 

The review board is made up of approximately thirty scholars from around the country 
(geography, gender and language are taken into consideration), with a few representatives 
outside of Canada. Reviewers are appointed for a period of two years, and are generally invited 
because they have reputations as excellent scholars and are expert in specific areas of interest 
to the journal. 

When a manuscript is received, it typically follows this route: 

• An administrative check is carried out to see whether the submission meets submission 
requirements (four copies, abstract, APA format). If the submission is complete, an 
acknowledgement card is sent to the author and the review process is initiated. If not, the 
author is sent a card requesting the missing items. 

• Editorial review. The paper is reviewed by the editor to determine whether it is appropriate 
for peer review. Some submissions might come from individuals unfamiliar with CJEC, so we 
occasionally received papers about bio-communication systems or accounting (both actual 
examples). Also, we rarely received a paper which was sloppy or hopelessly inept. In such cases, 
the editor may decide to reject the paper without subjecting it to a full review. 

• Reviewers are selected for the paper based on areas of expertise and availability (each 
reviewer receives only one manuscript at a time, except under unusual circumstances). 

• Peer Review. The author s name and affiliation is removed from the manuscript. The 
manuscript is then sent to three reviewers (two if three reviewers are not available). Each 
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reviewer receives a letter, review form and checklist with guidelines for review. Reviewers are 
asked to complete their reviews in four weeks. 

•Based on the re-views received and the editor's best instincts, a publication decision about the 
manuscript is made, 

•If accepted for publication, author(s) are notified of the due date for the final manuscript, 
given an outline of the required and suggested revisions, provided copies of the actual reviews, 
and sent routine permission forms. 

•When a revised manuscript is received, it is reviewed carefully to see that the changes made 
were true to the requirements laid out in your letter of acceptance. 

After the revised manuscripts are received, final decisions are made about the content of the 
issue. The design and layout are completed, and galleys are sent to the authors for a final 
reading and correction of any typographical mistakes. The entire process, from submission to 
final publication typically takes from six months to a year. 

Benefits of Scholarly Review 

I found many good points in the review procedures we followed, and the process of peer review 
generally performs a very real and important service to individuals and the field. It is not my 
intention in this paper to undermine the efforts of many dedicated scholars who give their 
time so unselfishly to their review tasks.-. 

One of the most important services provided by reviewers is the provision of valuable critical 
commentary on work. It has been my experience that drafts of manuscripts improve 
dramatically following thorough reviews and careful criticism. Reviewers are most often very 
deliberate and comprehensive in their reviews. In this way, peer review promotes both 
document and career development. 
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For the field of study, one of the most significant services provided by peer review is sifting the 
best material from the large amount of material available for publication. This, in turn, saves 
the readership a great deal of search time. If the best material has already been isolated, and 
available time for reading professional literature is limited, then the reader can readily select 
the "best" from the "best." Without question, there is some poorly conceived, pcoily executed 
and certainly poorly written material submitted for publication; many would argue that 
substandard material should not see the light of day, and peer review ensures that it seldom 
does. 

Tenure and piomotion committees in universities and various granting agencies require 
measures of the quality of work, and peer review provides a convenient method of 
discrimination. Most granting organizations, and university promotion committees ask 
applicants for a list of their refereed publications; few pay serious regard to non-refereed work. 
Indeed, peer review is often only the first criterion used for such purposes; the rejection rate of 
the journals in which refereed publications appear is also important. A severe rejection rate 
implies greater rigor to some groups. 

The sifting process also serves to establish professional standards for research and scholarly 
activity within a field of study. The private scrutiny of the review process is followed by the 
public display of selected works, and these set the standards against which other scholars 
compare their own work. Theoretically, this should constantly encourage scholars to 
improve, to measure up to and to exceed prevailing standards. For new scholars, such sifting 
not only sets standards of performance, it also identifies trends or areas of importance to the 
field of study. It identifies those areas of inquiry which are more likely to achieve publication. 

Perhaps one of the most important, but seldom discussed, arguments for rigorous review and 
high rates of rejection is economic. Professional societies and organizations usually subsidize 
the publication of academic journals; subscriptions alone seldom cover expenses. This 
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introduces the need to keep printing costs low, and places enormous pressure on most editors to 
work within strict page-count limits. Even if faced with nothing but outstanding submissions, 
a high percentage would have to be rejected in order to keep production costs in line with 
budgets. 

Drawbacks of Scholarly Review 

There are many negative points introduced by traditional approaches to academic publication, 
the first of which I would label reviewer bias. Reviewers are respected, successful members of a 
field of study. They have become successful in most instances by adopting the norms of their 
disciplines, and by building on well-established scholarly traditions and topics. This results 
in academic "regression toward the mean." Peer review, however well intended, discourages the 
exploration of fringe topics and frowns on radical departures from tradition. As a result, 
change can be painful and slow in some cases— witness the shift from objectivist to 
constructivist orientations in educational technology or the tension between quantitative and 
qualitative inquiry. Fifteen years ago, most established scholars in cur field we-e firmly 
rooted in a behavioristic tradition, with more liberal scholars considering the implications of 
cognitive psychology— both orientations clearly objectivist. Very few seriously considered 
constructivist philosophy, and qualitative methodologies rarely appeared in our journals, 
although several influential individuals were beginning to make noises in the field {cf. 
Lincoln and Guba, 1985). The movement toward qualitative inquiry and constructivist 
philosophy in our field of study has indeed been painful, and has sparked energetic debate in 
classrooms and conferences for many years. But our academic journals are only recently 
publishing qualitative research and articles about constructivism. Post-modern philosophy 
and deconstruction may offer the next field of battle for members of editorial review boards {cf 
Hiynka. D. and Beiland . 1991). 
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Research and peer review serve different masters. The purpose of research is to promote 
change; the major side-effect of peer review is inertia. 

Another type of bias is bias in favour of significant results. Research which reports no 
significant difference is not as likely to be published in most journals, even though such 
studies often signal important unproductive avenues of research. I have had the experience of 
mentioning a whopping NSD in a study I conducted to a colleague who quickly responded he 
had found the same thing. How many others in our field of study spend unproductive hours 
conducting studies they could have avoided if previously unsuccessful studies been readily 
available? As a community of scholars we are doomed to relive our failures if we do not expose 
our failures in our professional literature. 

Another drawback of our current approach to academic publishing is that some very good work 
either does not get published or its publication is seriously delayed. Most journals receive 
many more manuscripts than can be published in their available space. This means that 
worthy articles often go in search for an outlet— sometimes several outlets— before publication 
is achieved. This problem is exacerbated by the policy of most journals to only accept 
submissions which are not under review by other journals. The reason for this is 
understandable — a journal cannot use precious reviewer time or risk its prestige on a piece of 
work which may be pulled from the publication queue by an author in favour of another 
journal. At CJEC we require a letter from authors stating that a submission is not currently 
under review elsewhere before we will accept it for review. Because each review may take 
months to complete, material can become dated while searching for an appropriate journal. If 
good work is not published, the literature of our field of study suffers. If the body of literature 
from which we draw is diminished, so may the quality of future research be diminished. 
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Even if review and acceptance run smoothly, there are long delays from writing to 
publication. 2 I know of at least one journal which was notifying authors at the time of 
submission that the publication queue was up to two years (personal experience). This calls 
into question the currency of any research appearing in academic journals; books are even 
worse offenders. 

All editors share the common enemy of lack of consensus. Reviewers rarely agree, and it has 
been my experience that when they do, it is usually to condemn an article rather than praise it. 
This often puts the ultimate publication decision in the hands of the editor, and the editor can 
be influenced by other than purely academic matters (publication costs, available space, the 
reputation of the author). 

Blind review may be a problem for an editor drawn into a publication decision, but it is also a 
problem for reviewers. It is difficult to veil authors whose work is well-known or who 
repeatedly cite their own work in a manuscript. Even references to institutions can serve to 
influence reviewers; they may lean more favorably toward manuscripts submitted from 
prestigious institutions. Blind review is not always blind; in fact it often suffers only a mild 
sight impairment. 

My final criticism has less to do with academic issues than economics. The way we produce 
journals is unnecessarily expensive, environmentally harmful, and ultimately self-defeating. 
Producing paper copies of journals is expensive, and it places severe demands on the 
professional organizations which sponsor them. In addition, libraries must purchase and 
maintain expensive subscriptions. In my own university, and in many others around North 



These words, by the way. were written on August 9, 1993. 
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America, serials collections are under attack, and the high cost of producing print-based 
journals contributes to the problem. 

Some Suggestions 

The upshot of this discussion is that we need to find different ways to conduct the business of 
academic publishing. The first recommendation is that we move quickly to electronic formats 
for publication. We have the means to accomplish this, and certainly several journals are 
already moving in this direction. Elaborate computer-based electronic conferencing forums 
are available, and they have worked out many of the technical obstacles to electronic 
publication. 

Not only will this save natural resources (paper), it should mitigate the high costs of publishing 
and make material available to a wider audience. University consortia could support 
electronic journal systems by providing host computer space and network space. In Canada, 
the Social Sciences and Humanities Research Council could cooperate with research libraries 
around the country to establish network protocols using internet and other communication 
structures. 

Journals such as Nautilus have demonstrated exciting new avenues opened by multimedia 
electronic publishing. Traditional articles can incorporate windows of full motion video to 
illustrate points or demonstrate treatments. Interviews with authorities can include audio 
based versions of the interviews. The limitation at this point in time is that the CD-ROM 
technology needed to distribute multimedia is not universally available as of >et. But certainly 
it will be, or another method of dealing with multimedia data will emerge. 

But these are technical matters. There are powerful social consequences which accompany a 
change to electronic publication. For example, editors would not necessarily be restricted to 
publishing only a few articles, depending on the space made available by the printing budget. 
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Electronic space is inexpensive, so it would be technically feasible to publish everything 
submitted. 

Blasphemy! Publish everything? What would happen to the quality of our publications? 

I am not sure. Certainly the quality of a piece of work would not be presumed, as it can be 
merely by its appearance in a rigorous academic journal after running the gauntlet of blind 
peer review. Tenure and promotion committees and granting organizations would need to (and 
to be fair, some already do) emphasize other measures of the worth of publications. For 
promotion, samples of work are already routinely sent to expert external reviewers for 
assessment. Granting organizations also ask for comprehensive external reviews which 
include comments about the significance of the applicant's previous work. 

Certainly electronic publication can provide an open forum of debate, and promote a more 
democratic, less threatening environment in which academics can pursue their research 
interests. It may also stimulate more aggressive and daring research, and that, in my opinion, 
is sorely needed in our field of study. Work which deserves attention will get it: work which is 
substandard or of little interest will be given less attention over time. 

Another benefit of electronic publication will be the rapid introduction of material into the 
academic marketplace. Publication queues will largely disappear. 

One major challenge consumers will face is how to deal with the glut of information which will 
result from opening the l.'nes of publication. I often get frustrated when I conduct a literature 
search on a large data base such as the Educational Resources Information Center (ERIC). I 
seldom choose the optimal combination of descriptors to obtain a reasonable, yet not 
overwhelming, number of studies to review for any particular question. This problem will 
probably be worsened if the publishing gate is opened. A serious and important challenge for 



1.2 



Scholarly Review ac^. Academic Publication u 

information scientists and educational technologists will be to develop navigational 
approaches to facilitate academic review. 

I am not suggesting that all journals should move to electronic forms of publication 
immediately. I do, however, believe that electronic publication will soon become a significant 
alternative format for many types of publications, and education— particularly educational 
technology) should be at the forefront of such changes. A seemingly small c : ange such as this 
can have significant long-term consequences, and I hope the preceding discussion 
demonstrates that some of the most profound changes will occur in our own. academic 
subculture. 
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